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[57] ABSTRACT 

The present apparatus and method captures at least a pair of 
images in relatively quick succession with an illumination 
source being fired during the capture of one of the images. 
The illumination source is located adjacent to the optical 
lens or close to the optical axis of the camera. Next the 
illumination level of the images is compensated to make 
them equal in illumination. Following that an analysis of the 
images is performed to determine the difference between the 
illumination compensated images. After the illumination 
compensation the difference has to be caused by eye -defects 
or specular reflections from glasses. The pixel coordinates of 
the transitory eye-defect regions are recorded and analjrzed 
to determine the spacing of the potential eye locations and/or 
the spatial structure of the eyes for plausibility. Once con- 
firmed the coordinates are output and recorded. 

40 Claims, 5 Drawing Sheets 
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METHOD AND APPARATUS FOR separation, so that the illumination impinging on the sub- 

DETERMINING THE POSITION OF EYES jecls eyes is reflected at an angle that misses the taking lens. 

AND FOR CORRECTING EYE-DEFECTS IN A number of U.S. patents reflect the art prior to this 

A CAPTURED FRAME invention. Each wiU be discussed: 

HELD OF THE INVENTION ' U S^PaL No. 5^70,434, issued Oct. 29, 1996 to Badique, 

entitled "Circuit Arrangement For Recognizing A Human 

The invention relates generally to the field of digital Face", describes a circuit for recognizing a human face in a 

image processing, and in particular detecting human eyes or sequence of video images. The circuit includes the steps of 

even animal eyes in digital images. More specifically, the subtracting two consecutive image firames and using the 

invention relates to a method and apparatus for fast and areas of difference to determine if there is a moving object, 

accurate eye detection by taking advantage of the red-eye The difference map receives further post-processing, culmi- 

effect, in case human beings are captured by the apparatus. nating in a convolution to determine if a head and shoulders 

Animal eyes may be detected by taking advantage of the feature set is present in the scene. 

eye-defect in images captured together with a flash light. In y.S. Pat. No. 5,680,481, issued Oct 21, 1997 to Prasad et 

case of animal eyes the regions of an image showing the ^ ^1., entitled "Facial Feature Extraction Method and Appara- 

eye-defect are usually visible as bright green or yellow spots tus for a Neural Network Acoustic and Visual Speech 

in the captured image frame. Recognition System", describes another approach to facial 

Additionally, the invention relates to a method and appa- feature identification and extraction. In this patent, the 

ratus for fast and automatic correction of eye-defects in variation in gray scale tones firom a dull frontal face image 

captured image frames. are iised to locate the eyes and mouth by thresholding the 

_ ^ ^ ^ pixel values and finding centroids of the three areas. 

BACKGROUND OF THE INVENTION ^ ^ - \ . ^ , •« „ 

Many other prior art documents deal more speancally 

The increased use of computers in many applications has with eye or gaze tracking, or iris recognition methods or, 

drawn increasing focus on improving the man-machine ^5 face matching methods. 

interface. It is the desire of many applications to locate the u s. Pat, No. 5^25,133, issued Jun. 28, 1994 to Adachi, 

face of the user, then to process it to robusUy identify the entided "Device for Measuring a Retina Reflected Light 

person (for entitlement benefits recipients, for national bor- Amount and a Gaze Detecting Apparatus Using the Same" 

der crossings or secure area entry verification, as a replace- describes a device for measuring reflected Ught from a retina 

ment for the ubiquitous PIN numbers, etc.). The algorithms ^nd detecting the direction in which the user is looking with 

for facial recognition have dramaticaUy improved in recent ^n apparatus. This apparatus directs three sources of infrared 

years and are now sufficiently robust for many applications. emission, located at three different positions, toward a user's 

The weak part of the system is the face detection and system locates the pupils by pattern recognition or 

location front-end system. Other appHcations for facial red-eye ("detecting the frequency components corre- 

im aging beyond identification are growing in interest, in sponding to hemoglobin which is abundantly contained in 

particular perceptual computing, such as discerning a reac- t^e light reflected by the retina'*). It then processes the retinal 

tion or emotion from a user's face. This would enable reflection based on the displacement angles to determine 

computer-driven systems to be more responsive, like a where the user's gaze is directed. Thus, the apparatus 

human. Again, these algorithms wiU be limited by the functionality is keyed to computing the gaze angle from the 

weaknesses in face deteaion and location. ^ three dimensional angular measurements, instead of locating 

When flash illumination is used during the capture of an eyes. This art does not mention the issue of spacing between 

image that contains sizable human faces, the pupils of emission source and image capture device. It teaches away 

people sometimes appear red because the light is partially from the current invention by its dependence on measuring 

absorbed by capfllaries in the retina. As illustrated in FIG. 1, angular separation between emission source and image 

the light rays 70 from the flash illumination source 12 enter 45 pick-up source for a minimum of three emitter/detector 

the eye 2 through the eye lens 3, and form an image 12a of pairs. 

the illumination source 12 on retina 4. The eye^efect in u.S. Pat. No. 5,231,674 issued Jul. 27, 1993 to Cleveland 

captured images, known as the "red-eye effect" is mostly et al., entifled "Eye Tracking Method and Apparatus" 

seen with human eyes. In case animals are captured, the describes a system to determine the direction that a person 

eye-defect will show a bright green or yeUow color. Animal 50 is gazing, determining the point at which he is gazing, or 

eyes are generally more diflScult to detect for pattern rec- measuring the motion of his eye. The outputs of this 

ognition algorithms due to the large variations in animal apparatus are locations of eye features such as edge coor- 

facial structure, complexion, hair and structure of the eyes dinates between the pupil and iris of the eye and of the center 

itself. coordinates of light reflections off the cornea of the eye. An 

Referring now to FIG. 2, the light rays 80 reflected from 55 image intensity profile taken through a cross section of the 

the retina 4 exit the eye 2 through the eye lens 3, and finally eye iris, pupil (illuminated using the bright-eye effect and 

enter the camera lens 5. If the camera lens 5 is placed close the corneal reflection is extracted knowing the size of the 

to the illumination source 12, the red-eye effect will be eye, inferred from the distance between the eye and the 

maximized. In other words, the amoimt of red-eye or eye- cameras). Then, the extracted profile is smoothed. The peak 

defect being observed increases as the illumination source go region (defined by a set of thresholds) near the center of the 

12 gets closer to an optical axis 22 defined by the camera smoothed profile is detected as the corneal region. Typically 

lens 5 (see also FIG. 3). the relative intensities of the iris and the pupil are not 

The general technique for red-eye reduction in cameras sufficiently different to make the image processing easy, 

has been to impact two parameters: (a) Reduce the pupil Therefore, the bright-eye effect is used in this method and 

diameter of the subject, for example by emitting a series of 65 apparatus to increase the contrast ratio between the pupil and 

small pre-flashes prior to capturing the desired image with its surrounding iris. Consequently, the contrast ratio between 

fuU illumination; and, (b) Increase the flash to lens the iris and the bright pupil in Uie camera image can be made 
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significantly greater than the contrast ratio between the iris 
and a dark pupil. With the increased contrast ratio, image 
processing algorithms can locate the pupil edges and center 
more reliably and accurately for the particular eye gaze- 
tracking application. It would be noted that this method and 
apparatus assumes known eye location and eye size (on the 
camera optical axis and at a fixed distance) and is designed 
to track the eye gaze for speech impaired and physically 
handicapped persons. Moreover, this method and apparatus 
uses temporal changes in monochrome images and, thus, no 
color information. 

US. Pat. No. 5,432,863 issued Jul. 11, 1995 to Benati et 
al., entitled "Automated Detection and Correction of Eye 
Color Defects Due to Flash Illumination" describes a means 
to automatically detect and correct red-eye defects due to 
flash illumination in still images. The method includes 
means for determining whether a still image artifact is truly 
a red-eye problem based on shape, coloration, and bright- 
ness. 

SUMMARY OF THE INVENTION 
It is an object of this invention to provide a method for 
determining the position of eyes within a captured image 
frame and thereby overcoming one or more problems as set 
forth above. 

One of the preferred methods of the present invention is 
a method for determining the position of eyes within a frame 
of captured image data comprising the steps of: 
capturing two color frames of an image scene wherein one 

firame is captured with an ambient illumination and the 

other frame is captured with an additional illumination; 
storing said frames of said image scene in a memory; 
compensating said frames for the overall illumination 

change between said two captured frames to obtain a 

compensated frame; 
determining the difference between said compensated frame 

and an uncompensated frame of said two captured frames 

to obtain a compensated color difference image; 
scanning said compensated color difference image and 

determining pairs of regions of high intensity pixel value 

in at least one color channel; and 
outputting the data of pixel pair coordinates wherein each 

coordinate corresponds to a center of each region of high 

intensity pixel value in said color channel within said 

captured image scene. 

A further object of this invention is to provide an appa- 
ratus capable of performing each of the above steps. An 
apparatus according to the invention comprises: 
means for capturing at least two color frames of an image 
scene, said means is provided with an optical lens defining 
an optical axis; 
means for storing the data of at least two consecutive 

captured color frames of an image scene; 
an illumination source for illuminating one of said captured 
color frames of said image scene wherein said illumina- 
tion source is positioned as close as possible to said 
optical axis; 
electronic means 

a) for compensating said frames for the overall illimiina- 
don change between said two captured frames to obtain 
a compensated frame; 

b) for determining the difference between said compen- 
sated frame and an uncompensated of said two captured 
frames to obtain a compensated color difference image; 

c) for scanning said compensated color difference image 
and determining pairs of regions of high intensity pixel 
value in at least one color channel; and 
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d) for outputting the data of pixel pair coordinates 
wherein each coordinate corresponds to a center of 
each region of high intensity pixel value in said color 
channel within said captured image. 
5 This invention describes electronics for imaging a con- 
sumer's face and providing an output video stream as well 
as the coordinates of the eyes, if a face is present in the 
image frame. This invention will describe the application of 
a unique combination of conventional technology to deliver 
10 inexpensively and robustly the eye coordinates. 

It is however, the novelty of this invention to take 
advantage of the physical cause of the red-eye effect and use 
it to facilitate fast and accurate eye detection. Accurate eye 
detection will enable enhancement and manipulation of 
15 images containing one or more human faces. For example, 
red-eye correction can be rehably performed. 

A further object of the invention is to describe a method 
for a digital camera capable of locating eye-defects in a 
captured image, in frame and either removing the eye- 
20 defects in the camera or recording the locations of the 
eye-defects in an image frame to enable eye-defect removal 
in the photo-finishing stage. Another method of the present 
invention is a method used in a digital camera capable of 
locating eye-defect regions comprising the steps of: 
25 selecting a function of the digital camera for eye-defect free 
picture taking; 

capturing two color frames of an image scene with a 
minimum delay in between to minimize spatial registra- 
tion error wherein one frame is captured with an ambient 
30 illumination and the other frame is captured with an 
additional camera triggered illmnination; 

storing data, representing said two captured frames in a 
memory; 

compensating said frames for the overall illumination 
35 change between said two captured frames to obtain a 

compensated frame; 
determining the difference between said compensated frame 

and an uncompensated frame of said two captured frames 

to obtain a compensated color difference image; 
40 scanning said compensated color difference image and 

determining pairs of regions of high intensity pixel value 

in at least one color channel; and 
obtaining the data of pixel pair coordinates wherein each 

coordinate corresponds to a center of each of said eye- 
45 defect regions. 

A further object of this invention is to provide a digital 
camera capable of performing each of the above steps. A 
digital camera according to the invention comprises: 
a user input control for initiating the eye-defect correction 
5Q function of the digital camera; 

means for capturing two color frames of an image scene with 

a minimum delay between to minimize spatial registration 

error, wherein said capturing means is provided with an 

optical lens defining an optical axis; 
55 an iUimiination source for illuminating one of the two 

frames of said image scene wherein said illumination 

source is positioned as close as possible to said optical 

axis; 

means for storing the data of said two captured frames of 
50 said image scene; and 
electronic means 

a) for compensating said frames for the overall illximina- 
tion change between said two captured frames to obtain 
a compensated frame 
65 b) for determining the difference between said compen- 
sated frame and an uncompensated of said two captured 
frames to obtain a compensated color difference image; 
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c) for scanning said compensated color difference image recording means for writing the spatial coordinates of the 

and thereby determining pairs of regions of eye-defect, eye-defects in said recording layer on said photographic 

and calculating spatial coordinates of said eyc-dcfccts. film inside said photographic film camera. 

An additional object of the invention is to describe a These and other aspects, objects, features, and advantages 

method implemented in a photographic film camera which is 5 of the present invention will be more clearly understood and 

capable of recording the locations of the detected eye- appreciated from a review of the following detailed descrip- 

defects on the photographic film inside the camera. The tion of the preferred embodiments and appended claims, and 

removal of the detected eye-defects is done m the photofin- reference to the accompanying drawing?, 
ishing stage. This object is accomphshed by a method used 

in a photographic film camera capable of locating eye-defect ADVANTAGEOUS EFFECT OF THE 

regions comprising the steps of: INVENTION 
selecting a function of the photographic film camera for 

eye-defect free picture taking; The present invention has the following advantages: 

exposing the photographic fihn with a frame of an image technique quickly determines the location of eyes in an 

scene bearing the camera triggered Ulumination and inexpensive and robust manner; and 

simultaneously storing data of said frame bearmg the ^ . . . . 1 j r . u * t. 

camera triggeJed Uluiiination in a memory, 2. technique enables eye^iefect fitee photography, 

capturing a further frame of said image scene with a mini- DESCRIFHON OF THE DRAWINGS 

mum delay m between to minunize spatial registration 

error, wherein said frame is captured with an ambient pjc. 1 is an illustration of how the light rays from an 

illumination, and storing data representing said frame in 20 illumination source get into individuals eyes and forai an 

said memory; image of the illumination source on the retina; 

compensating said frames for the overall illumination _ . n * c u *u r u* ♦ a 

change be^een said two captured frames to obtain a ^^^'^ ^. «° ^^J'^^on of how the hght rays reflec ed 

compensated frame- retina get into a camera s lens mounted close to the 

determining the difference between said compensated frame 25 ill^™i°ation source; 

and an uncompensated frame of said two captured frames FIG. 3 is an illustration of an eye detection apparatus; 

to obtain a compensated color difference image; fig. 4 is an illustration of an eye-defect free digital 

scanning said compensated color difference image and camera* 

determining pairs of regions of high intensity pixel value p, '5 ^ iuustration of an cye^efect free photographic 

m at least one color channel; in fil • i- e> i- 

obtaining data of pixel pair coordinates corresponding to the ^™ camera, 

pairs of region of high intensity value wherein each FIG. 6 is a block flow diagram representing the operating 

coordinate corresponds to a center of each of said eye- functionality of the apparatus as disclosed in the embodi- 

defect regions; and ment of FIG. 3; 

recording on the photographic film inside the camera coor- piG. 7 is a block flow diagram representing the operating 

dinates of said pairs of regions of high intensity pixel ^5 functionality of the camera embodiments shown in FIGS. 4 

value preferable in the red channel. jjjj 5. 

A further object of this invention is to provide a photo- ' ^ . ui 1 « a- f ^ ^ 

graphicfilmcaieracapableofperformingeachoftheabovc . 8 is a block flow diagram of the eye location 

steps. Aphotographic fihn camera according to the invention confirmation operaUons. 

comprises: 40 To fadlilate understanding of the invention, identical 

a user input control for initiating the eye-defect free funaioo reference numerals have been used, where possible, to 

of the fihn camera; designate identical elements that are common to the figures. 

means for capturing two frames of an image scene with a nccr^DiPTinM np thf 

minimum delay between to minimize spatial registration DETAILED DESCRIPTION OF THE 

error, wherein said captiiring means is provided with an 45 INVENTION 

optical lens defining an optical axis; configuration of an apparatus for determining the 

an illumination source for illuminating one of said two position of eyes within a captured image frame is depicted 

frames of said image scene, wherein said illumination ^ pjQ 3 jhis system can be used for a number of 

source is positioned as close as possible to said optical apphcations, including user identification (e.g., for ATM), 

^is; 50 user monitoring (e.g., for checking the alertness of car 

a photographic fihn having a recording layer for capturing drivers), and pose optimization for ID photo taking (with 

said illuminated frame of said image scene; ^udio feedback instructions generated by the photo- taking 

means for temporarily storing data of said illuminated frame system when red-eye effect is maximized). In particular, eye 

and data of a frame bearing ambient illumination of said location can be used to perform spatial normalization of the 

image scene; 55 f for face recognition, model-based face image 

electronic means compression, etc. In addition, eye movement and iris motion 

a) for compensating said frames for the overall illumina- can be monitored to check if a driver is still alert. The 
tion change between said two capmred frames to obtain apparatus 10 includes an iUumination source 12 which is 
a compensated frame, attached to means 14 for capturing at least two image frames 

b) for determining the difference b between said compen- eo of an image scene. In the embodiment as shown in FIG. 3, 
sated frame and an uncompensated frame of said two the means 14 for capturing is a video camera. A driver circuit 
captured frames to obtain a compensated color differ- 16 is attached to the video camera and a frame grabber 18 
ence image; is positioned in the video camera. Additionally the video 

c) for scanning said compensated color difference image camera is provided with an optical lens 20 which defines an 
and thereby determining pairs of regions of eye-defect, 65 optical axis 22. The video camera is connected by an electric 
and calculating spatial coordinates of said eye-defects; cable 24 to electronic means 26. The electronic means 
and comprises a monitor 28, a computer 30, a keyboard 32, a 
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Storage or a memory 34, and a printer 36. The illumination 
source 12 may include a lamp or an infra-red firing device, 
power supply and triggering circuit (not shown). A key 
element of the configuration is to use the smallest spacing 
between the illumination source 12 and the optical lens 20. 
In other words, the illumination source 12 should be placed 
as close to the camera's optical axis 22 as possible. This will 
give the narrowest retinal reflection angle and thus the 
strongest eye-defect to make a robust pupil location feature. 
On the contrary, common practice in conventional camera 
design is to make the angle between these components as 
large as possible to minimize the reflection which causes an 
eye-defect in images. 

The present invention utilizes the red-eye phenomenon to 
serve as a feature for locating the coordinates of human eyes 
within a captured frame of an image scene. On the otber 
hand even animals' eyes can be located within an image 
scene. In this case the eye -defect is not a red-eye effect, as 
with human eyes, instead it is of green or yellow color. Id 
order to detect animals* eyes within an image scene, the 
apparatus 10 has to scan the image for bright regions in the 
green channel. 

Referring now to FIG. 6, there is shown a block flow 
diagram for the basic operational functionality for eye 
coordinate detection using the apparatus of FIG. 3. The 
initial assumption is that the video camera 14 is live, 
capturing images when the face coordinate detection ensues. 
After pressing start S20 first, a color frame of video with 
ambient illumination is saved S21 to the storage 34. Next, 
the illumination source 12 is triggered so that light is emitted 
and the next color video frame S22 is captured with an 
additional illumination. The captured frames are compen- 
sated S23 for the overaU illumination change between the 
two captured frames to obtain a compensated frame. The 
compensated frame is subtracted S24 from an uncompen- 
sated frame of the two captured frames to obtain a color 
compensated difference image. The xmcompensated frame 
can be either the frame bearing the ambient illumination or 
the frame bearing the additional illumination. The compen- 
sated color difference image is scanned for any pairs of 
regions that have high intensity pixel values, particularly in 
at least one color channel S25, especially when the location 
of human eyes are determined, the method scans for pairs of 
regions that have high intensity pixel values, particularly in 
the red channel. In the case that animal eyes are to be located 
in a captured image frame, the method scans for regions of 
high intensity values particularly in the green channel. The 
high intensity region pairs may be further processed to 
determine if they have the expected color, size, shape, 
structure, and spacing to plausibly represent the eye loca- 
tions on a human face S26. If these conditions are satisfied, 
the pixel pair coordinates of the centers of the regions of 
high intensity pixel vahie corresponding to the centers of the 
pupil regions within the image frame arc recorded together 
with the captured image S27. If the conditions are not 
completely satisfied, alternative error correcting measures 
may be taken. If no high intensity regions in the compen- 
sated difference image are detected, that state is indicated 
S29 to the electronic means 26. If the conditions are partially 
satisfied, that state is indicated to the computer 30, which 
will send control signals to ±e driver circuitry 16 to capture 
another set of images and test again S28. 

Referring now to FIG. 4, the function of eye-defect free 
picmre taking can be enabled by a user input control 42 
which is part of a digital camera 40. The digital camera 40 
comprises an illiunination source 12 and electronic means 
26. The elertronic means 26 includes a central processing 



unit 44 and a memory 46. For capmring the individual 
frames of an image scene, the digital camera 40 is provided 
with an optical lens 20 which defines an optical axis 22. The 
optical lens 20 and an image taking sensor 48 are aligned on 

S the optical axis 22. 

The operation steps are illustrated in FIG. 7. After initia- 
tion S30 of the process, two color frames of an image scene 
are captured with minimum delay in between to minimize 
optical registration error. One frame S32 is captured with an 

IQ ambient illumination and the other frame is captured S31 
with an additional camera triggered illumination. 

These two images are stored in the memory 46 where 
software or firmware residing in the camera imemory 46 
performs the brightness compensation S33, differencing and 

15 confirmation operations in the central processing unit 44. A 
compensated color difference image is obtained by deter- 
mining S34 the difference between the compensated image 
and an uncompensated image. The compensated color dif- 
ference image is then scanned S35 for regions of high 

20 intensity pixel value in at least one color channel. As 
mentioned above, the eye-defect to be detected can be the 
red-eye effect, known from human eyes or a yellow-eye or 
green-eye effect as known from animal eyes. In order to 
detect the red-eye effect, the compensated color difference 

25 image has to be scanned for regions of high intensity pixel 
value in the red channel. To locate animal eyes within an 
image frame, the compensated color difference image has to 
be scanned for regions of high intensity pixel value in the 
green channel. Once the data of the pixel pair coordinates 

30 arc obtained, and the detected region docs satisfy S3 6 aU of 
the eye criteria, the eye-defect correction may be performed 
automatically in the digital camera 40. Furthermore, the user 
can select to record S37 the spatial coordinates of the 
eye-defect regions along with the captured eye-defect image 

35 in the memory 46 of the digital camera. The correction for 
the eye-defect is performed in the photofinishing lab. In case 
the detected eye regions do not satisfy all of the eye criteria 
S38, the detection procedure will be started again. An error 
code will be outputted if none of the eye criteria is detected 

40 S39. 

Referring now to FIG. 5, the function of eye-defect free 
picture taking is implemented in a photographic film camera 
50. Auser can enable the function by a user input control 42. 
The photographic film camera 50 comprises an illumination 

45 source 12, which in one embodiment is a flash light emitting 
unit. AdditionaUy, electronic means 26 are provided, which 
include a camera built in central processing unit 52, a 
memory 54 for temporary storing and recording means 56. 
The recording means are designed to record information on 

50 a photographic film 58 inside the camera 50. For capturing 
the image frames the camera 50 is provided with an optical 
lens 20 which defines an optical axis 22. The photographic 
film 58 and the optical lens 20 are aligned on the optical axis 
22. Prior to the photographic fihn 58, a shutter 60 is located 

55 which allows the exposure of the photographic film 58. 
Additionally the camera 50 has a built-in image sensor 62 
which is located aside from the optical axis 22. A semi- 
transparent mirror 64 is tilted at an angle of 45"* with respect 
to the optical axis 22 in order to direct light from the image 

60 scene simultaneously onto the photographic fikn 58 and onto 
the image sensor 62. 

The operation steps of the photographic film camera 50 
are also illustrated in FIG. 7. After selecting the function for 
eye-defect free picture taking, the camera will capmre an 

65 image scene and thereby expose the photographic film 58 
with a frame of the image scene bearing the camera triggered 
illumination. Parallel to the exposure of the photographic 
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film 58, the image sensor 62 is exposed with the same frame Then a color difference image is obtained by taking the 

and the data representing the frame are stored in the memory difference between the compensated non-illuminated frame 

54. A fiirther frame of the image scenes is captured with a and the frame bearing the additional illumination. Again, the 

minimum delay in between to minimize spatial registration selected sequence should not be regarded as a limitation, 
error. This frame is captured with an ambient illumination 5 

and the Kght coming from this frame is used solely to expose />(^;',A)=i..(^>'.*)-{ii(^y.AHA^.(*)-Af,C*)]} Equation 3 

the image sensor 62. The data representing the frame is ^^^^^ ^ ^^^^^^ difference in the kth color 

stored also in the memory 54. Software or firmware residmg component for a pixel at the coordinate of (x, y). 
in said memory 54 of said photographic film camera 50, ^^^^ ^^^^^ multiple people at different distances or 
performs the brightness compensation, differencmg and 10 angles with respect to the iUumination source, the illumina- 
confirmation operations in the central processing umt 52. A ^^^^^^ ^ uniform across the image. In this case, the 

detailed description of the process performed m the central compensation in Equation 3 should be performed based on 

processing unit 52 is already mentioned in the paragraph j^^^ estimates of the log image, 
describing the process in the digital camera 40. 

After the electronic means 26 in the photographic film ^5 ^ V Equation 4 

camera 50 has located and determined the pairs of regions of ^* ^"^^^1^ ^ 

high intensity pixel value in at least one color channel, the ^y«vuj^> 
data corresponding to the center of each of the eye-defect 1 ^ 

region is recorded on the photographic film 58 inside the Miix.y.^)^ ^ Zj^ ^.{x,y,k) 

camera 50. In one embodiment the photographic film 58 and 20 . ^ 

the camera 50 are a film and camera from the Advanced 

Photo System. The recording of the data is a magnetic where Mi(x, y, k) denotes the local mean of the kth color 

recording in the magnetic layer of the exposed photographic component of L^, and N(x, y) denotes a local window 

fihn 58. The photographic film camera 50 provides record- centered around the current pixel at the coordinate of (x, y), 
ing means 56 which can record in the magnetic layer of the ^ ^-^e of the local window, 

photographic film 58. The following rules can be used to locate the pair of pupil 

The following description is limited to one procedure of regions (the rules are used for detecting the eye-defect 

capturing frames of an image scene. The first picture or known as the red eye-effect): 
image frame is captured with the camera triggered 

niumination, and the second picture or image frame is i)(^>ircd)>axZ>(^y,bluc) Equations 

captured with an ambient illumination. Nevertheless, the z)(x,y,red)>axZ)(;»;y,green) Equation 6 

above is not considered as a limitation since it is obvious to 

any person skilled in the art to carry out the above procediu-e D{x,y,ttd)>^ Equation? 

in that the first captured image frame bears the ambient . « « , ^ . • . • , 

-11 TO,' „ *u A ^ V 35 where a is set to 2.0 and B IS set to 50 m this embodiment 

illummation. This method or procedure is particularly ^ , , . ^ , ^ . • • i 

advantageous for a digiul camera or a film camera: (1) the «=r=nce miagc converted back to the original 

spatial registration error is minimized to benefit the differ- video metnc. He operaUons defined by equations 5 to 7 can 

endng operation; (2) the flash unit of the camera will not be performed m the log image metnc. EquaUons 5 and 

recha^ed in time for second firing; and (3) the non-flash ^ ensure that there is significant higher mciease in the red 

image is captured and bufEeied only as needed. « componeot relaUve to the other color components. Equation 

^ . 7 ensures that the absolute mcrease m red component is 

More specificaUy the illummaUon compensation step S23 j^ficant. The coordinates of the pupils, as weU as their 

(see FIG. 6), or 833 (see FIG 7) is performed m the extracted and stored. If multiple faces are present 

following fashion. The followmg description shows, that the ^ . • detected. In case 

first image is capUired with ambient illummation and the ^^^^ eye-defects have to be detected or located than the red 

second image is captured together with a camera tnggercd eye-effect (e.g. for "green eyes" or "yellow eyes" in 

flluminatioD. Each image is converted mto a log unage using animals), the equations 5 through 7 have to be amended with 

Equation 1: respect to the green color channel In principle, this method 

i,(x,>vi)=iog(/,(^;^Jt>+i) t)e generalized to detect any significant local changes in 

50 one or more color components. 

L^{x,y,k)~iogUx,y,k)^i) EquaUon 1 particular, the eye confirmation step S26 (see FIG. 6), 

where Ii(x,yjc) denotes the intensity vahie of the kth color or S36 (see FIG. 7) includes the foUowing steps as disclosed 

component of a pixel at the coordinate of (x,y), and Li(x,y,k) ^ FIG. 8. This step is mainly used to ensure the presence of 

denotes the log-transformed intensity value. Other variables eye in order to limit false positive detection. For each 

are defined in analog. Then, the global mean of each log 55 extracted pupil pair, the size and orientation of the eyes are 

image is computed: estimated based on the spacing of the pair. The estimated 

size is checked with respect to the pupil size S41. Next, a 

1 Equation 2 t»lock of image is cut out after adjusting for size and 

2^^*^*' ^'^^ tiquation orientation with respect to a pre-defined eye template S42. 

60 A plurality of such blocks of image can be extracted within 
1 ^ a small search window around the initial pupil position S43. 
M2ik)=^lj^ix^y^k) ^ cross-correlation score is computed between the pre- 
defined template and an image block centered within the 
small search window S44. Then, a local peak of cross- 
where Mi(k) denotes the global mean of the kth color 65 correlation is located within the search window S45. Note 
component of Lj and Z is the size of the image. Other that the center of the pupil may coincide with the center of 
variables are defined in analogy. the eye in some cases. Therefore, depending on each 
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application, a refined eye location can be obtained based on 
the local peak of cross-correlation. For red-eye removal, the 
center of the pupil is of interest. On the other hand, for 
spatial normalization of the face or pose estimation, the 
center of the eye is more desirable. If the local peaks do not 
have a cross-correlation score above a pre-defined threshold 
S46, an appropriate error code is output S49. Otherwise, the 
eye pair is further checked using a set of eye-pair criteria to 
ensure the presence of eyes. In particular, an intensity profile 
that goes through the centers of both eyes is extracted and 
smoothed. This profile is checked for the typical peaks and 
valleys that correspond to the main structure of eyes and the 
nose ridge between the eyes S47. If the eye-pair criteria arc 
satisfied, the eye location is confirmed S48. Otherwise, an 
appropriate error code is output S49. 

The illumination source 12 can be a flash unit mounted on 
or built into the film or digital camera. For an apparatus such 
as illustrated in FIG. 3, an infra-red (IR) illumination source 
can be used. For certain digital or film cameras, an IR source 
is used for auto-focusing and therefore can be used for 
red-eye generating and detection. 

The invention has been described with reference to pre- 
ferred embodiments. However, it will be appreciated that 
variations and modifications can be effected by a person of 
ordinary skill in the art without departing fi-om the scope of 
the invention. 
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What is claimed is: 

1. A method for determining the position of eyes within a 
frame of captured image data comprising the steps of: 
capturing two color fi'ames of an image scene wherein one 

frame is captured with an ambient illumination and the 

other frame is captured with an additional illumination; 
storing said firames of said image scene in a memory; 
compensating said frames for the overall illumination 

change between said two captured frames to obtain a 

compensated frame; 
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detentnining the difference between said compensated 
frame and an uncompensated fiiime of said two cap- 
tured frames to obtain a compensated color difference 
image; 

5 scanning said compensated color difference image and 
determining pairs of regions of high intensity pixel 
value in at least one color channel; and 
outputting the data of pixel pair coordinates wherein each 
coordinate corre^onds to a center of each region of 

JO high intensity pixel value in said color channel within 
said captured image scene. 

2. The method according to claim 1, wherein said com- 
pensating step comprises: 

a) converting the value of each pixel in said captured 
frames into log metric; 

b) calculating the mean log intensity value of each of said 
converted images; 

c) adding the difference between said mean log intensity 
values to the converted image corresponding to the 
subtrahend of said difference; and 

20 

d) converting said compensated log image to the inverse 
log metric to obtain a compensated frame. 

3. The method according to claim 1, wherein said deter- 
mined pairs of regions of high intensity pixel value, are 
processed in the red channel for human eyes, according to 
color, size, shape structure, and spacing. 

4. The method according to claim 1, wherein said deter- 
mined pairs of regions of high intensity pixel value, are 
processed in the green channel for animal eyes, according to 
color, size, shape structure, and spacing. 

5. The method according to claim 1, wherein said deter- 
mined pairs of regions of high intensity pixel value, pref- 
erably in the red and green chaimels for human and animal 
eyes, are processed according to color, size, shape structure, 
and spacing. 

6. The method according to claim 1 fiirther comprising the 
step of: analyzing the spacing of a potential eye location and 
the spatial structure for plausibility. 

7. The method according to claim 1 wherein the output- 
^ ting step comprises recording the pixel coordinates of the 

center of said regions of high intensity pixel value. 

8. The method according to claim 1 wherein said frames 
of said image scene are consecutive picmres of a video 
image stream. 

9. Ad apparatus for determining the position of eyes 
within a frame of captured image data comprising: 

means for capturing at least two color frames of an image 
scene, said means is provided with an optical lens 
defining an optical axis; 
50 means for storing the data of at least two consecutive 
capmred frames of an image scene; 
an illumination source for illuminating one of said cap- 
tured frames of said image scene wherein said illumi- 
nation source is positioned as close as possible to said 
55 optical axis; and 
electronic means: 

a) for compensating said frames for the overall illumi- 
nation change between said two captured frames to 
obtain a compensated frame; 
60 b) for determining the difference between said com- 
pensated frame and an uncompensated frame of said 
two captured frames to obtain a compensated color 
difference image; 
c) for scanning said compensated color difference 
65 image and determining pairs of regions of high 

intensity pixel value in at least one color channel; 
and 
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d) for outpulting the data of pixel pair coordinates 
wherein each coordinate corresponds to a center of 
each region of high intensity pixel value in said color 
channel within said captured image scene. 

10. Apparatus according to claim 9 wherein the capturing 
means is a video camera for capturing consecutive frames of 
an image scene. 

11. Apparatus according to claim 9 wherein the illtmiina- 
tion source is a flash light emitting source for triggering 
direct illumination in synchronization with the capturing of 
the second image frame of said image scene. 

12. i^paratus according to claim 9 wherein the electronic 
means is a computer to which a keyboard, a monitor, an 
external storage, and a printer are connected. 

13. A method, used in a digital camera, for locating 
eye-defect regions comprises the steps of: 

selecting a function of the digital camera for eye-defect 
free picture taking; 

capturing two color frames of an image scene with a 
minimum delay in between to minimize spatial regis- 
tration error, wherein one frame is captured with an 
ambient illumination and the other frame is captured 
with an additional camera triggered illumination; 

storing data, representing said two captured frames in a 
memory; 

compensating said frames for the overall illumination 
change between said two captured frames to obtain a 
compensated frame; 

determining the difference between said compensated 
frame and an uncompensated frame of said two cap- 
tured frames to obtain a compensated color difference 
image; 

scanning said compensated color difference image and 
determining pairs of regions of high intensity pixel 
value in at least one color channel; and 

obtaining the data of pixel pair coordinates of said regions 
of bright intensity pixel value wherein each coordinate 
corresponds to a center of each of said eye-defect 
regions. 

14. The method according to claim 13, wherein said 
compensating step comprises: 

a) converting the value of each pixel in said captured 
frames into log metric; 

b) calculating the mean log intensity value of each of said 
converted images; 

c) adding the difference between said mean log intensity 
values to the converted image corresponding to the 
subtrahend of said difference; and 

d) converting said compensated log image to the inverse 
log metric to obtain a compensated frame. 

15. The method according to claim 13 further comprising 
the step of: analyzing the spacing of a potential eye-defect 
region and the spatial structure for plausibility. 

16. The method according to claim 13 wherein the out- 
putting step comprises recording the pixel coordinates of the 
center of said eye-defect regions. 

17. The method according to claim 15, wherein said 
determined eye- defect regions of high intensity pixel value, 
are processed in the red channel for human eyes, according 
to color, size, shape structure, and pacing. 

18. The method according to claim 15, wherein said 
determined eye- defect regions of high intensity pixel value, 
are processed in the green channel for animal eyes, accord- 
ing to color, size, shape stmcture, and spacing, 

19. The method according to claim 15, wherein said 
determined eye- defect regions of high intensity pixel value. 
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preferably in the red and green channels for human and 
animal eyes, are processed according to color, size, shape 
structure, and spacing. 

20. Method according to claim 13 wherein based on the 
user selection the digital camera performs an automatic 
eye-defect correction of the captured image scene. 

21. Method according to claim 13 wherein, based on the 
user selection, the digital camera stores the data of the frame 
bearing said camera triggered illumination along with the 
obtained ^atial data of the centers of each of said eye-defect 
regions. 

22. A digital camera capable of locating eye-defect 
regions comprising: 

a xiser input control for initiating the eye-defect correction 
ftinction of the digital camera; 

means for capmring two color frames of an image scene 
with a minimum delay between to minimize spatial 
registration error, wherein said capturing means is 
provided with an optical lens defining an optical axis; 

an illumination source for illuminating one of the two 
frames of said image scene wherein said illumination 
source is positioned as close as possible to said optical 
axis; 

means for storing the data of said two captured frames of 

said image scene; and 
electronic means: 

for compensating said frames for the overall illumina- 
tion change between said two captured frames to 
obtain a compensated frame; 

for determining the difference between said compen- 
sated frame and an uncompensated frame of said two 
captured frames to obtain a compensated color dff- 
ference image; and 

for scanning said compensated color difference image 
and thereby determining pairs of regions of eye- 
defect, and for calculating spatial coordinates of said 
eye-defects. 

23. The digital camera according to claim 22 wherein said 
means for storing is used to store the captured image having 
the eye-defect correction automatically performed. 

24. The digital camera according to claim 22 wherein said 
means for storing is used to store the data of the frame 
bearing said camera triggered illumination along with the 
calculated data of the spatial coordinates of the centers of 
each of said eye-defect regions. 

25. >^)paratus according to claim 22 wherein the illumi- 
nation source is a flash light emitting source. 

26. i^paratus according to claim 22 wherein the elec- 
tronic means is a built-in central processing unit which 
controls the functions of the digital camera and an attached 
memory. 

27. A method, used in a photographic film camera, for 
locating eye-defect regions comprising the steps of: 

selecting a function of the photographic film camera for 
eye-defea free picture taking; 

exposing the photographic film with a frame of an image 
scene bearing the camera triggered illumination and 
simultaneously storing data of said frame bearing the 
camera triggered illumination in a memory; 

capturing a further frame of said image scene with a 
minimum delay in between to minimize spatial regis- 
tration error, ^^ile^ein said frame is capmred with an 
ambient illumination, and storing data representing said 
frame in said memory; 

compensating said frames for the overall illumination 
change between said two captured frames to obtain a 
compensated frame; 



03/31/2004, EAST Version: 1.4.1 



6,134,339 



15 



16 



detennining the difference between said compensated 
frame and an uncompensated frame of said two cap- 
tured frames to obtain a compensated color difference 
image; 

scanning said compensated color difference image and 
detennining pairs of regions of high intensity pixel 
value in at least one color channel; 

obtaining data of pixel pair coordinates corresponding to 
the pairs of regions of high intensity pixel value 
wherein each coordinate corresponds to a center of 
each of said eye-defect regions; and 

recording on the photographic film inside the camera 
coordinates of said pairs of regions of high intensity 
pixel value preferably in the red charmel. 

28. The method according to claim 27, wherein said 
compensating step comprises: 

converting the value of each pixel in said captured frames 

into log metric; 
calculating the mean log intensity value of each of said 

converted images; 
adding the difference between said mean log intensity 

values to the converted image corresponding to the 

subtrahend of said difference; and 
converting said compensated log image to the inverse log 

metric to obtain a compensated frame. 

29. The method according to claim 27 further comprising 
the step of analyzing the spacing of a potential cye-defecl 
region and the spatial structure for plausibility. 

30. The method according to claim 29, wherein said 
determined eye-defect regions of high intensity pixel value, 
are processed in the red channel for human eyes, according 
to color, size, shape structure, and spacing. 

31. The method according to claim 29, wherein said 
determined eye -defect regions of high intensity pixel value, 
are processed in the green channel for animal eyes, accord- 
ing to color, size, shape structure, and spacing. 

32. The method according to claim 29, wherein said 
determined eye-defect regions of high intensity pixel value, 
preferably in the red and green channels for human and 
animal eyes, are processed according to color, size, shape 
structure, and spacing. 

33. Method according to claim 27 wherein based on the 
user selection, the digital camera performs an automatic 
eye-defect correction of the captured image scene. 

34. Method according to claim 27 wherein based on the 
user selection the digital camera stores the data of the frame 
bearing said camera triggered illumination along with the 
obtained spatial data of the centers of each of said eye-defect 
regions. 

35. The method according to claim 27, wherein said film 
includes a magnetic layer and, wherein the recording on the 
photographic film is magnetic recording on said magnetic 
layer. 
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36. A photographic film camera capable of locating eye- 
defects comprises; 

a user input control for initiating the eye-defect free 
function of the film camera; 
^ means for capturing two frames of an image scene with a 
minimum delay between to minimize spatial registra- 
tion error, wherein said capturing means is provided 
with an optical lens defining an optical axis; 
an illumination source for illuminating one of said two 
frames of said image scene, wherein said illumination 
source is positioned as close as possible to said optical 
axis; 

a photographic film for capturing said illuminated frame 
15 of said image scene, said film including a recording 
layer; 

means for temporarily storing data of said illuminated 
frame and data of said frame bearing ambient illumi- 
nation of said image scene; 
20 electronic means 

for compensating said frames for the overall illumina- 
tion change between said two captured frames to 
obtain a compensated frame; 
for determining the difference between said compeo- 
25 sated frame and an uncompensated frame of said two 

captured frames to obtain a compensated color dif- 
ference image; 
for scanning said compensated color difference image 
and thereby determining pairs of regions of eye- 
30 defect, and calculating spatial coordinates of said 

eye-defects; and 
recording means for writing the spatial coordinates of the 
eye-defects in said recording layer on said photo- 
graphic film inside said photographic film camera. 

37. The photographic film camera according to claim 36 
wherein the capturing means comprise a semi transparent 
mirror located in said optical axis in order to simultaneously 
illuminate said photographic film and an image sensor, said 
illumination of said photographic film is controlled by a 

^ shutter, positioned prior to said photographic film. 

38. The photographic film camera according to claim 36 
wherein the recording layer on said photographic film is a 
magnetic recording layer and the recording means are 
designed for magnetic recording of data on said magnetic 
recording layer. 

39. The photographic film camera according to claim 36 
wherein the illumination source is a flash light emitting 
source. 

40. The photographic film camera according to claim 36 
wherein said electronic means is a camera built-in central 
processing unit which controls the main camera functions, a 
memory for temporary storing and said recording means. 
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